Touchscreen hover detection in an augmented and/or virtual reality environment

ABSTRACT

A system for detecting and tracking a hover position of a manual pointing device, such as finger(s), on a handheld electronic device may include overlaying a rendered mono-chromatic keying screen, or green screen, on a user interface, such as a keyboard, of the handheld electronic device. A position of the finger(s) relative to the keyboard may be determined based on the detection of the finger(s) on the green screen and a known arrangement of the keyboard. An image of the keyboard and the position of the finger(s) may be rendered and displayed, for example, on a head mounted display, to facilitate user interaction via the keyboard with a virtual immersive experience generated by the head mounted display.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.62/219,423, filed Sep. 16, 2015, the disclosure of which is incorporatedherein by reference.

FIELD

This document relates, generally, to detection and tracking of anelectronic device in an immersive augmented reality and/or virtualreality system.

BACKGROUND

An augmented reality (AR) and/or a virtual reality (VR) system maygenerate a three-dimensional (3D) immersive environment. A user mayexperience this 3D immersive virtual environment through interactionwith various electronic devices, such as, for example, a helmet or otherhead mounted device including a display, glasses or goggles that a userlooks through when viewing a display device, gloves fitted with sensors,external handheld devices that include sensors, and other suchelectronic devices. Once immersed in the virtual environment, userinteraction with the virtual environment may take various forms, suchas, for example, physical movement and/or physical interaction withvirtual elements in the virtual environment, and/or manipulation of anelectronic device to interact with, personalize and control the virtualenvironment.

SUMMARY

In one aspect, a method may include generating, by a head mountedelectronic device operating in an ambient environment, a virtualenvironment, generating, within a display of a handheld electronicdevice operating in the ambient environment, a mono-chromatic screenoverlaid on a user interface within the display of the handheldelectronic device, while maintaining touchscreen functionality of theuser interface, detecting, by a camera of the head mounted electronicdevice, a physical object in a foreground of the mono-chromatic screen,determining, by a processor of the head mounted electronic device, aposition of the detected physical object relative to the user interfacebased on a detected position of the physical object in the foreground ofthe mono-chromatic screen, and displaying, by the head mountedelectronic device, a virtual image of the user interface and a virtualimage of the detected physical object on a display of the head mountedelectronic device.

In another aspect, a system may include a head mounted electronic deviceincluding a display, a camera and a processor, the head mountedelectronic device configured to be operably coupled with a handheldelectronic device, wherein the camera is configured to capture an imageof the handheld electronic device, and the processor is configured tosegment an image of a physical object detected between the head mountedelectronic device and the handheld electronic device, and to display avirtual image in a virtual scene generated by the head mountedelectronic device based on the image captured by the camera, the virtualimage displayed in the virtual scene including a virtual image of thedetected physical object, and a virtual image of a user interface of thehandheld electronic device, the user interface being overlaid by amono-chromatic screen rendered on the handheld electronic device, in avirtual scene generated by the head mounted electronic device, based onthe image captured by the camera.

In another aspect, a computer program product embodied on anon-transitory computer readable medium may include, stored thereon, asequence of instructions which, when executed by a processor causes theprocessor to execute a method, including generating a virtualenvironment in a head mounted electronic device operating in an ambientenvironment, capturing, by a camera of the head mounted electronicdevice, an image of a handheld electronic device operably coupled withthe head mounted electronic device, detecting, in the image captured bythe camera, a physical object in a foreground of a mono-chromatic screenoverlaid on a user interface on the handheld electronic device, the userinterface maintaining functionality beneath the mono-chromatic screen,separating the detected physical object from the image captured by thecamera and generating an image of the separated physical object,detecting a position of the detected physical object relative to theuser interface based on a detected position of the physical object inthe foreground of the mono-chromatic screen, and displaying a virtualimage in the virtual environment generated by the head mountedelectronic device, the virtual image including a virtual image of theuser interface and a virtual image of the separated physical objectseparated from the image captured by the camera

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of an augmented and/or virtual reality systemincluding a head mounted display device and a handheld electronicdevice, in accordance with implementations described herein.

FIGS. 2A and 2B are perspective views of an example head mounteddisplay, in accordance with implementations described herein.

FIGS. 3A and 3B are front views of an example handheld electronicdevice, in accordance with implementations described herein.

FIG. 4 is a block diagram of a first electronic device and a secondelectronic device of an augmented and/or virtual reality system, inaccordance with implementations described herein.

FIGS. 5A-5E illustrate detection and tracking of a position of a manualpointing device relative to a user interface of a handheld electronicdevice, and display of manual inputs received at the user interface, ona head mounted electronic device, in accordance with implementationsdescribed herein.

FIG. 6 is a flowchart of a method of detecting and tracking a hoverposition of a pointing device relative to a user interface of a handheldelectronic device in an augmented and/or virtual reality system, inaccordance with implementations described herein.

FIG. 7 shows an example of a computer device and a mobile computerdevice that can be used to implement the techniques described herein.

DETAILED DESCRIPTION

A user immersed in a virtual environment, for example, a 3D augmentedand/or virtual reality environment, wearing, for example, a head mounteddisplay (HMD) device may explore the virtual environment and interactwith the virtual environment through, for example, physical interaction(such as, for example, hand/arm gestures, head movement, walking and thelike), and/or manipulation of the HMD, and/or manipulation of a separateelectronic device, to experience the immersive virtual environment. Forexample, in some implementations, the HMD may be paired with a handheldelectronic device, such as, for example, a controller, a smartphone orother such handheld electronic device. User manipulation of the handheldelectronic device paired with the HMD may allow the user to interactwith the virtual environment generated by the HMD, and/or may allow theuser to make use of other functionality of the handheld electronicdevice while immersed in the virtual environment.

In some implementations, a user may manipulate the handheld electronicdevice paired with the HMD to cause a desired action in the virtualenvironment generated by the HMD. In some situations, because a displayof the HMD is displaying the virtual environment, and/or one or morevirtual objects, it may be difficult for the user to provide some typesof inputs on the handheld electronic device, as it may be difficult forthe user to view the handheld electronic device, and in particular, itmay be difficult for the user to manipulate a user interface of thehandheld electronic device while wearing the HMD. For example, it may bedifficult for a user to make a text entry using a keyboard userinterface of the handheld electronic device without being able to seethe keyboard on the handheld electronic device and/or a finger positionrelative to the keyboard on the handheld electronic device.

To address at least some of the issues noted above, in someimplementations, a mono-chromatic screen, such as a green screen, may berendered and/or overlaid on a display of the handheld electronic device,and in particular, overlaid on a display of a user interface on thehandheld electronic device, so that a hover position of the user'shand/fingers may be detected by, for example a camera of the HMD.Touchscreen functionality of the user interface may be maintained withinthe display area of the handheld electronic device, even though themono-chromatic screen is overlaid on the user interface and thus is notactually visible on the display. The user interface of the handheldelectronic device may be virtually rendered and displayed in the virtualenvironment generated by the HMD for viewing and use by/interaction withthe user. Accordingly, the user may be able to physically interact withthe user interface of the handheld electronic device and make use of thefunctionality of the user interface within the display area of thehandheld electronic device, even though the user interface is notvisible to the user on the handheld electronic device, but instead isvirtually displayed in the virtual environment generated by the HMD. Theuser may accurately interact with the user interface and make use of thefunctionality provided by the user interface based on the virtual userinterface that is virtually displayed within the virtual environmentgenerated by the HMD.

In some implementations, an image of a finger hover position, detectedbased on an image capture of the user's finger against the monochromaticscreen overlaid on the user interface of the handheld electronic device,may be virtually displayed, together with the virtual rendering of theuser interface, for example, a virtual keyboard, in the virtualenvironment generated by the HMD, so that the user may view the virtualrendering of the keyboard and relative finger position on the keyboardwithout having direct physical visibility of the keyboard displayed onthe handheld electronic device, to facilitate text entry. In someimplementations, the rendering of a mono-chromatic screen, or greenscreen, for detection of finger hover position may allow, for example, asimple RGB camera of the HMD to accurately detect and track fingerposition, essentially in real time, in a relatively efficient andaccurate manner. The detected finger position may be virtually displayedin the virtual environment generated by the HMD (for example, togetherwith the user interface) as a pass through image captured by the cameraof the HMD, as a virtual image rendered by, for example, a processor ofthe HMD based on the image captured by the camera of the HMD, and thelike.

In the example implementation shown in FIG. 1, a user wearing an HMD 100is holding a portable handheld electronic device 102, such as, forexample, a controller, a smartphone, or other portable handheldelectronic device which may be operably coupled with, or paired with theHMD 100 for interaction in the virtual environment generated by the HMD100. In the example shown in FIG. 1, the user is holding the handheldelectronic device 102 in a portrait orientation, in his right hand.However, the user may also hold the portable electronic device 102 withonly his left hand, or both his right hand and his left hand, and/or ina landscape orientation, depending on a particular input interface,input mode, and other such factors, and still interact with the virtualenvironment generated by the HMD 100. To address at least some of theissues noted above, in some implementations, a mono-chromatic screen,such as a green screen, or other color screen, may be rendered and/oroverlaid on a display of the handheld electronic device, and inparticular, overlaid on a display of a user interface on the handheldelectronic device, so that a hover position of the user's hand/fingersmay be detected by, for example a camera of the HMD.

FIGS. 2A and 2B are perspective views of an example HMD, such as, forexample, the HMD 100 worn by the user in FIG. 1 to generate an immersivevirtual environment to be experienced by the user. The HMD 100 mayinclude a housing 110 in which optical components may be received. Thehousing 110 may be coupled, for example, rotatably coupled and/orremovably attachable, to a frame 120 which allows the housing 110 to bemounted or worn on the head of the user. An audio output device 130 mayalso coupled to the frame 120, and may include, for example, speakersmounted in headphones and coupled on the frame 120. In FIG. 2B, a frontface 110 a of the housing 110 is rotated away from a base portion 110 bof the housing 110 so that some of the components received in thehousing 110 are visible. A display 140 may be mounted on the front face110 a of the housing 110. Lenses 150 may be mounted in the housing 110,between the user's eyes and the display 140 when the front face 110 a isin the closed position against the base portion 110 b of the housing110. A position of the lenses 150 may be may be aligned with respectiveoptical axes of the user's eyes to provide a relatively wide field ofview and relatively short focal length.

In some implementations, the HMD 100 may also include a sensing system160 including various sensing system devices and a control system 170including various control system devices to facilitate operation of theHMD 100. The control system 170 may also include a processor 190operably coupled to the components of the control system 170.

In some implementations, the HMD 100 may also include a camera 180 whichmay capture still and/or moving images of the ambient, or physical, orreal world environment, outside of the HMD 100 and the virtualenvironment generated by the HMD 100. For example, in someimplementations, the camera 180 may capture an image, or series ofimages, of the user's finger(s) and/or hand(s) relative to a surface,for example, a display surface on which a user interface may bedisplayed, of a handheld electronic device, such as, for example, thehandheld electronic device 102 shown in FIG. 1, operably coupled withthe HMD 100. In some implementations, the camera 180 may capture stilland/or moving images of elements in the real world environment in whichthe augmented and/or virtual reality system operates.

These images of the real world environment, including the images of theuser's finger(s) and/or hand(s) relative to the handheld electronicdevice 102, may be displayed to the user on the display 140 of the HMD100 in a pass through mode. In some implementations, this may allow theuser to view a pass through image of the captured elements of the realworld environment, for example, a pass through image of the user'sfinger(s) and/or hand(s) overlaid on a virtual rendering of the userinterface, such as for example, the keyboard, in the virtualenvironment. In some implementations, this may allow the user totemporarily leave the virtual environment and return to the real worldenvironment without removing the HMD 100 or otherwise changing theconfiguration of the HMD 100 to move the housing 110 out of the line ofsight of the user. In some implementations, the camera 180 may be adepth camera that can determine a distance from the camera 180 on theHMD 100 to, for example, the user's hand(s) holding the handheldelectronic device 102, based on the relatively consistent infrared (IR)response of skin.

As shown in FIG. 3A, in some implementations, the handheld electronicdevice 102 may include an audio output device 103, such as, for example,a speaker, an audio input device 104, such as, for example, amicrophone, an imaging device 105, such as, for example, a camera, and adisplay 106 displaying images on the handheld electronic device 102. Insome implementations, the display 106 may be a touch sensitive display106 so that the display 106 may output images, and may also output auser interface including regions designated to receive a user input viaa touch sensitive surface of the touch sensitive display 106. In someimplementations, the handheld electronic device 102 may render agraphical user interface on the touch sensitive display 106, such as,for example, a keyboard 107 or other type of user interface, configuredto receive user touch inputs. The user interface can be configured sothat interactions with the user interface can trigger functionalityassociated with physical regions of a display area of the display 106.

In some implementations, the handheld electronic device 102 may render amono-chromatic screen, or a green screen overlay 108 on the display 106of the handheld electronic device 102, as shown in FIG. 3B. The greenscreen overlay 108 can be displayed within a display area of the display106 while the underlying functionality associated with the userinterface, for example as shown in FIG. 3A, is maintained. With thegreen screen overlay 108 on the display 106, a hover position of theuser's finger(s) relative to the touch sensitive display 106, andrelative to individual input elements of the graphical user interface,such as the keyboard 107, may be quickly and accurately detected by, forexample, the camera 180 of the HMD 100 (while still maintaining theunderlying functionality of the user interface). Said differently, insome implementations, the graphical user interface, such as the keyboard107, may remain active while covered, or overlaid by the green screenoverlay 108, so that a user input, such as a touch detected on the touchsensitive surface of the handheld electronic device 102, may generate acorresponding user input even though the graphical user interface, orkeyboard 107, is not visible on the display 106 of the handheldelectronic device 102 due to the rendering of the green screen overlay108. The user may be able to accurately interact with the userinterface, or keyboard 107, because the user interface and/or handheldelectronic device 102 may be rendered and displayed to the user withinthe virtual environment generated by the HMD 100.

In some situations, the green screen overlay 108 on the display 106 mayprovide a measure of security when the user interacts with the userinterface, such as the keyboard 107, to enter sensitive information suchas, for example, login information including usernames, passwords andthe like. That is, in this arrangement, the keyboard 107, and entriesmade using the keyboard 107, are only visible to the user, and not toother parties, as the user interface, such as the keyboard 107, remainsfunctional at the touch sensitive surface of the display 106, but is notvisible to other parties at the display 106 of the handheld electronicdevice 102 because of the green screen overlay 108.

The example user interface 107 displayed on the display 106 of thehandheld electronic device 102 shown in FIG. 3A is a keyboard, that canreceive a user's touch input and translate that touch input into a textinput in, for example a text entry field displayed on the display 106 ofthe handheld electronic device 102, and/or the in virtual environmentgenerated by the HMD 100. A keyboard is shown as an example of the userinterface in FIG. 3A, and described hereinafter, simply for ease ofexplanation and illustration. The user interface may include numerousother types of user interfaces, including other features and elements,in addition to or instead of, a keyboard, such as, for example,selection buttons, slide buttons, menu items, and the like.

Similarly, the user interface 107 displayed on the display 106 of thehandheld electronic device 102 shown in FIG. 3A is a keyboard renderedwith the handheld electronic device 102 oriented in the landscapeposition, in which the device 102 may be held in both the left and righthands of the user for text entry with finger(s) of both the left andright hands. However, the user interface 107, in the form of a keyboardor other user interface based on a particular application, may also bedisplayed with the handheld electronic device 102 oriented in theportrait position, in which the device 102 may be held in one of theleft hand or the right hand of the user, or both the left and righthands of the user, for text entry with finger(s) of the one of the lefthand or the right hand, or both the left and right hands. In either thelandscape orientation or the portrait orientation, the user interfacemay include numerous other features, in addition to or instead of, akeyboard, such as, for example, selection buttons, slide buttons, menuitems, and the like

A block diagram of a system for detecting and tracking a hover positionof a manual pointing device, such as a user's hand and/or finger, withrespect to a handheld electronic device in a virtual reality environmentis shown in FIG. 4. The system may include a first electronic device 400in communication with a second electronic device 402. The firstelectronic device 400 may be, for example an HMD as described above withrespect to FIGS. 1 and 2A-2B, generating a virtual environment, and thesecond electronic device 402 may be, for example, a handheld electronicdevice such as the handheld electronic device described above withrespect to FIGS. 1 and 3A-3B, which may communicate with the HMD tofacilitate user interaction with the virtual environment generated bythe HMD. For example, as described above, manipulation and/or physicalmovement of the second (handheld) electronic device 402 may betranslated into a desired interaction in the virtual environmentgenerated by the first (head mounted) electronic device 400.

The first electronic device 400 may include a sensing system 460 and acontrol system 470, which may be similar to the sensing system 160 andthe control system 170, respectively, shown in FIGS. 2A and 2B. In theexample implementation shown in FIG. 4, the sensing system 460 mayinclude numerous different types of sensors, including, for example, alight sensor, an audio sensor, a distance/proximity sensor, and/or othersensors and/or different combination(s) of sensors. In someimplementations, the light sensor, image sensor and audio sensor may beincluded in one component, such as, for example, a camera, such as thecamera 180 of the HMD 100 shown in FIGS. 2A and 2B. The control system470 may include numerous different types of devices, including, forexample, a power/pause control device, audio and video control devices,an optical control device, a transition control device, and/or othersuch devices and/or different combination(s) of devices. In someimplementations, one module of a video control device of the controlsystem 470 may be configured to generate and display one or more userinterfaces as described above, and another module of the video controldevice may be configured to generate and display a mono-chromaticscreen, or green screen, as described above, or a single module of thevideo control device of the control system 470 may be configured togenerate and display the one or more user interfaces and the greenscreen. In some implementations, the sensing system 460 and/or thecontrol system 470 may include more, or fewer, devices, depending on aparticular implementation. The elements included in the sensing system460 and/or the control system 470 can have a different physicalarrangement (e.g., different physical location) within, for example, anHMD other than the HMD 100 shown in FIGS. 2A and 2B.

The first electronic device 400 may also include an audio output device430 and an image output device 440, which may be similar to, forexample, the audio output device 130 and the display 140 shown in FIGS.2A and 2B. The audio output device 430 and the image output device 440may output audio and video signals related to the virtual immersiveexperience generated by the first electronic device 400. The firstelectronic device 400 may also include a processor 490 in communicationwith the sensing system 460 and the control system 470, a memory 480accessible by, for example, a module of the control system 470, and acommunication module 450 providing for communication between the firstelectronic device 400 and another, external device, such as, forexample, the second electronic device 402 paired with the firstelectronic device 400.

The second electronic device 402 may include a communication module 406providing for communication between the second electronic device 402 andanother, external device, such as, for example, the first electronicdevice 400 operably coupled with, or paired with the second electronicdevice 402. The second electronic device 402 may include a sensingsystem 407 including, for example, an image sensor and an audio sensor,such as is included in, for example, a camera and microphone, aninertial measurement unit, and other such sensors and/or differentcombination(s) of sensors. A processor 409 may be in communication withthe sensing system 407 and a controller 405 of the second electronicdevice 402, the controller 405 having access to a memory 408 andcontrolling overall operation of the second electronic device 402. Thesecond electronic device 402 may also include an audio output device 403and an image output device 404, which may be similar to, for example,the audio output device 103 and the display 106 shown in FIGS. 3A-3B.The audio output device 403 and the image output device 404 may outputaudio and image signals which, in some implementations, may provide aninterface for user input and interaction with the second electronicdevice 402 and/or with the first electronic device 400 operably coupledwith, or paired with, the second electronic device 402.

In some implementations, a text entry, or other input made by the uservia the user interface 107 as described herein, may include, forexample, an input of login information, an input related to instructionsto be carried out in the virtual environment such as characterselections, equipment selections, environment selections and the like,and other such inputs. In some situations, a user immersed in thevirtual environment may receive, for example, an incoming phone call ortext message which may temporarily interrupt the user's interaction inthe virtual environment generated by the HMD 100. FIGS. 5A-5E illustratean example in which an incoming text message alert is received while theuser is immersed in an ongoing virtual experience, simply for ease ofdiscussion and illustration. However, as noted above, the process to bedescribed herein may be applied to numerous other instances in which aninput, for example, a text input, a selection, a slide input, or othertype of input, may be made by the user via a user interface on ahandheld electronic device.

An example of a virtual scene 500, as viewed by the user, for example,on the display 140 of the HMD 100, is shown in FIG. 5A. FIG. 5A alsoillustrates the user physically holding the handheld electronic device102, which may be operably coupled with the HMD 100, for interaction inthe virtual environment generated by the HMD 100. While the user isimmersed in the virtual environment, for example immersed in the virtualscene 500, generated by the HMD 100, an incoming text message may bereceived by the handheld electronic device 102. In this example, analert 502 in the form of a visual indicator may be generated, alertingthe user to the incoming text message. The alert 502 may be displayed onthe display 106 of the handheld electronic device 102. However, becausethe user is wearing the HMD, the display 106 of the handheld electronicdevice 102, and the alert 502, may not be directly visible to the user.Thus, a virtual rendering of the alert 502, or a corresponding virtualalert 502V, may be displayed in the virtual scene 500 viewed by theuser, as shown in FIG. 5B, so that the virtual alert 502V is visible tothe user without removing the HMD 100. A mono-chromatic screen 108, suchas, for example, a green screen 108, may be rendered and overlaid on theuser interface of the handheld electronic device 102 (e.g., the physicaldevice), and in particular the green screen 108 may be overlaid on thedisplay 106 of the handheld electronic device 102 on which the alert 502is displayed, the alert 502 providing for user selection of a “Y” buttonor an “N” button indicating whether or not the user wants to retrievethe incoming message.

In some implementations, the mono-chromatic screen 108 may be generatedand overlaid on the user interface of the handheld electronic device 102in response one or more triggering events. Triggering event may include,for example, a user alert anticipating a user input and/or response viathe user interface on the handheld electronic device 102, anticipationof user input based on user interaction with a virtual object in thevirtual environment, movement of the user and/or the handheld electronicdevice 102 to indicate input via the user interface, and othersituations related to user input via the user interface in whichtracking of the user's finger/hand position relative to the userinterface will facilitate input.

As shown in FIG. 5C, the green screen 108 may be overlaid on the display106 of the handheld electronic device 102, including the area of thedisplay 106 at which the alert 502 would be displayed/otherwise visibleon the physical device 102, while the virtual alert 502V is displayed inthe virtual scene 500 as viewed by the user. That is, the user may view(e.g., virtually view) the virtual scene 500 on the display of the HMD100 including the virtual alert 502V and the selection buttons “Y” and“N” which the user may select to indicate whether or not the userintends to retrieve the incoming message. As described above, theoverlay of the green screen 108 on the user interface of the handheldelectronic device 102 (e.g., the physical device) may allow the camera180 of the HMD 100 to quickly and accurately capture, detect and track aposition of the user's finger relative to the handheld electronic device102, and in particular a position of the user's finger(s) relative tothe user interface (in this example, the alert 502 and selection buttonsY and N).

An image of the user's finger positioned relative to the virtual userinterface, or virtual alert 502V and selection buttons Y and N, may bedisplayed to the user in the virtual scene 500, as shown in FIG. 5C. Theimage of the user's finger relative to the user interface may becaptured by, for example, the camera 180 on the HMD 100, and a virtualimage of the user's finger relative to the virtual user interface (forexample, the virtual alert 502V) may be displayed in the virtual scene500 as a pass through image captured by the camera 180. In someimplementations, the image of the user's finger relative to the virtualuser interface may be rendered by, for example the processor of the HMD100 based on the image captured by the camera 180 of the HMD 100, andthe virtual rendering of the user's finger may be displayed in thevirtual scene 500. In some implementations, the image of the user'sfinger relative e to the virtual user interface in the virtual scene 500In some implementations, the image of the user's finger and the userinterface (in this example, the virtual alert 502V) may be overlaid onthe current virtual scene 500, as shown in FIG. 5C, or may replace thecurrent virtual scene 500 in the user's field of view.

Upon retrieving the incoming message, as shown in FIG. 5D, a textmessaging session may proceed between the user and the sender of theincoming message. The text messaging session may be visible to the uservia the virtual scene 500 virtually displayed to the user, for example,on the display 140 of the HMD 100. As shown in FIG. 5D, to provide fortext entry, the keyboard 107 may be made available on the user interfaceof the physical handheld electronic device 102, and the green screen 108may be overlaid on the user interface, or keyboard 107, of the physicalhandheld electronic device 102, to facilitate the detection and trackingof finger position relative to the user interface by the camera 180 ofthe HMD 100 as described above.

As shown in FIG. 5D, in some implementations, while the green screen 108is rendered and is overlaid on (or replaces) the keyboard 107 generatedby the handheld electronic device 102, a corresponding virtual keyboard107V may be rendered in the virtual scene 500 displayed to and viewed bythe user on the HMD 100, so that a rendering of the keyboard 107 on thehandheld electronic device 102 and a finger position with respect to thekeyboard 107, in the form of the virtual keyboard 107V in the virtualscene 500, may be viewed by the user via the display 140 of the HMD 100.As noted above, a hover position of the user's finger relative to thekeyboard 107 on the handheld electronic device 102 may be captured by,for example, the camera 180 of the HMD 100. Because the image of theuser's finger is captured against the green screen 108 overlaid on thedisplay 106 displaying the user interface, or keyboard 107, on thehandheld electronic device 102, a position of the finger may beaccurately and efficiently detected, tracked and displayed (either as apass through image or as a rendered image), essentially real time. Theessentially real time detection, tracking and display of the fingerposition against the green screen 108, with the user interface in theform of the keyboard 107 in this example, may allow the user to view avirtual visual representation of the entry of text in the virtual scene500, without removing the HMD 100 and/or terminating the virtualimmersive experience to obtain visibility of the physical handheldelectronic device 102. Because the user interface (and functionalityassociated therewith), in the form of the keyboard 107 on the physicalhandheld electronic device 102 in this example, remains active whileoverlaid by the green screen 108, the handheld electronic device 102 mayregister a user input in response to an actual touch (e.g., physicalcontact) on the touch sensitive surface of the display 106 of thehandheld electronic device 102 as discussed above. Upon completion ofthe messaging session shown in FIG. 5D, the user may resume activity inthe virtual environment, with minimal interruption, as shown in FIG. 5E.

The phrase green screen has been used throughout to refer to keying, inwhich a subject in the foreground may be detected and tracked against abackground defined by a single color screen, or mono-chromatic screen,the single color of the mono-chromatic screen not duplicated by thesubject in the foreground. Although green is used as the example colorin the implementations of a mono-chromatic screen as described herein,any single color may be used for the overlay rendered by the handheldelectronic device, as long as that single color is not duplicated in theitem to be tracked (in this case, the user's hand(s) and/or finger(s))and there is adequate color separation between the subject in theforeground (in this case, the user's hand(s) and/or finger(s)) and thebackground. For example, in some implementations, the mono-chromaticscreen may be rendered in, for example, a blue or grey or black hue, orany other single color that provides for adequate color separation.

A method 600 of detecting a hover position of a user's hand(s) and/orfinger(s) relative to a handheld electronic device in an augmentedreality and/or a virtual reality system, in accordance withimplementations described herein, is shown in FIG. 6. As noted above,the handheld electronic device may be similar to the handheld electronicdevice 102 shown in FIGS. 1 and 3A-3B, and may be paired with, forexample, an HMD such as, for example, the HMD 100 as shown in FIGS. 1and 2A-2B, to generate an immersive virtual environment to beexperienced by the user. The handheld electronic device 102 may bepaired with, and/or communicate with, the HMD 100 by, for example, via awired connection, a wireless connection via for example wifi orBluetooth, or other type of connection. After the HMD 100 and thehandheld electronic device 102 have been activated and paired, at block610, and an immersive virtual reality experience has been initiated atblock 620, a camera, such as, for example, the camera 180 of the HMD100, or other camera within the system that is capable of capturing animage of the user's hand(s) and the handheld electronic device 102, maybe activated to capture an image of the handheld electronic device 102at blocks 640 and 650.

As noted above, the image captured by the camera 180 of the HMD 100 mayinclude an image of the finger(s) and/or hand(s) of the user against themono-chromatic screen, or green screen 108, rendered and overlaid on theuser interface, for example, on the keyboard 107 displayed on thedisplay 106 of the handheld electronic device 102. If, at block 650, amanual pointing device, such as, for example, a finger, is detectedagainst the mono-chromatic screen 108 in the image captured by thecamera 180, the image of the manual pointing device may be segmentedfrom the background defined by the mono-chromatic screen 108, at block660, and a position of the manual pointing device may be correlated withat least one element of the user interface overlaid by themono-chromatic screen 108, at block 670. For example, at least aposition of a finger detected within the captured image, against thegreen screen 108 may be correlated with at least one key of the keyboard107 that has been overlaid by the green screen 108 on the display 106 ofthe handheld electronic device 102.

An image of the manual pointing device on the user interface at thecorrelated position, such as, for example, the finger at a positioncorresponding to one of the keys of the keyboard 107, may be segmentedfrom the image captured by the camera 180, at block 680, and the imageof the finger may be displayed in the virtual scene generated by the HMD100, for example, relative to the virtual keyboard 107V displayed in thevirtual scene 500 by the HMD 100, at block 685. In some implementations,the image of the manual pointing device on the user interface at thecorrelated position (i.e., the finger at a position corresponding to oneof the keys of the keyboard 107) may be rendered, for example, by aprocessor of the HMD 100, and displayed in the virtual scene, at anappropriate position with respect to the virtual keyboard 107V displayedin the virtual scene 500. In this manner, the user may view a positionof the finger on the virtual keyboard 107V, corresponding to thedetected position of the finger relative to the keyboard 107 of the userinterface of the physical handheld electronic device 102, without thephysical handheld electronic device 102 being directly visible to theuser, thus facilitating manual entry of user inputs through a touchscreen surface of the handheld electronic device 102 in an augmentedand/or virtual reality environment.

This process may be repeatedly performed until it is determined, atblock 690, that the virtual immersive experience has been terminated.

FIG. 7 shows an example of a generic computer device 800 and a genericmobile computer device 750, which may be used with the techniquesdescribed here. Computing device 800 is intended to represent variousforms of digital computers, such as laptops, desktops, tablets,workstations, personal digital assistants, televisions, servers, bladeservers, mainframes, and other appropriate computing devices. Computingdevice 750 is intended to represent various forms of mobile devices,such as personal digital assistants, cellular telephones, smart phones,and other similar computing devices. The components shown here, theirconnections and relationships, and their functions, are meant to beexemplary only, and are not meant to limit implementations of theinventions described and/or claimed in this document.

Computing device 700 includes a processor 702, memory 704, a storagedevice 706, a high-speed interface 708 connecting to memory 704 andhigh-speed expansion ports 710, and a low speed interface 712 connectingto low speed bus 714 and storage device 706. The processor 702 can be asemiconductor-based processor. The memory 704 can be asemiconductor-based memory. Each of the components 702, 704, 706, 708,710, and 712, are interconnected using various busses, and may bemounted on a common motherboard or in other manners as appropriate. Theprocessor 702 can process instructions for execution within thecomputing device 700, including instructions stored in the memory 704 oron the storage device 706 to display graphical information for a GUI onan external input/output device, such as display 716 coupled to highspeed interface 708. In other implementations, multiple processorsand/or multiple buses may be used, as appropriate, along with multiplememories and types of memory. Also, multiple computing devices 700 maybe connected, with each device providing portions of the necessaryoperations (e.g., as a server bank, a group of blade servers, or amulti-processor system).

The memory 704 stores information within the computing device 700. Inone implementation, the memory 704 is a volatile memory unit or units.In another implementation, the memory 704 is a non-volatile memory unitor units. The memory 704 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 706 is capable of providing mass storage for thecomputing device 700. In one implementation, the storage device 706 maybe or contain a computer-readable medium, such as a floppy disk device,a hard disk device, an optical disk device, or a tape device, a flashmemory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 704, the storage device 706,or memory on processor 702.

The high speed controller 708 manages bandwidth-intensive operations forthe computing device 700, while the low speed controller 712 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In one implementation, the high-speed controller 708 iscoupled to memory 704, display 716 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 710, which may acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 712 is coupled to storage device 706 and low-speed expansionport 714. The low-speed expansion port, which may include variouscommunication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)may be coupled to one or more input/output devices, such as a keyboard,a pointing device, a scanner, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 700 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 720, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 724. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 722. Alternatively, components from computing device 700 may becombined with other components in a mobile device (not shown), such asdevice 750. Each of such devices may contain one or more of computingdevice 700, 750, and an entire system may be made up of multiplecomputing devices 700, 750 communicating with each other.

Computing device 750 includes a processor 752, memory 764, aninput/output device such as a display 754, a communication interface766, and a transceiver 768, among other components. The device 750 mayalso be provided with a storage device, such as a microdrive or otherdevice, to provide additional storage. Each of the components 750, 752,764, 754, 766, and 768, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 752 can execute instructions within the computing device750, including instructions stored in the memory 764. The processor maybe implemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor may provide, for example,for coordination of the other components of the device 750, such ascontrol of user interfaces, applications run by device 750, and wirelesscommunication by device 750.

Processor 752 may communicate with a user through control interface 758and display interface 756 coupled to a display 754. The display 754 maybe, for example, a TFT LCD (Thin-Film-Transistor Liquid Crystal Display)or an OLED (Organic Light Emitting Diode) display, or other appropriatedisplay technology. The display interface 756 may comprise appropriatecircuitry for driving the display 754 to present graphical and otherinformation to a user. The control interface 758 may receive commandsfrom a user and convert them for submission to the processor 752. Inaddition, an external interface 762 may be provide in communication withprocessor 752, so as to enable near area communication of device 750with other devices. External interface 762 may provide, for example, forwired communication in some implementations, or for wirelesscommunication in other implementations, and multiple interfaces may alsobe used.

The memory 764 stores information within the computing device 750. Thememory 764 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 774 may also be provided andconnected to device 750 through expansion interface 772, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 774 may provide extra storage space fordevice 750, or may also store applications or other information fordevice 750. Specifically, expansion memory 774 may include instructionsto carry out or supplement the processes described above, and mayinclude secure information also. Thus, for example, expansion memory 774may be provide as a security module for device 750, and may beprogrammed with instructions that permit secure use of device 750. Inaddition, secure applications may be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 764, expansionmemory 774, or memory on processor 752, that may be received, forexample, over transceiver 768 or external interface 762.

Device 750 may communicate wirelessly through communication interface766, which may include digital signal processing circuitry wherenecessary. Communication interface 766 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 768. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 770 mayprovide additional navigation- and location-related wireless data todevice 750, which may be used as appropriate by applications running ondevice 750.

Device 750 may also communicate audibly using audio codec 760, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 760 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 750. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 750.

The computing device 750 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 780. It may also be implemented as part of a smartphone 782, personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made without departingfrom the spirit and scope of the invention.

In addition, the logic flows depicted in the figures do not require theparticular order shown, or sequential order, to achieve desirableresults. In addition, other steps may be provided, or steps may beeliminated, from the described flows, and other components may be addedto, or removed from, the described systems. Accordingly, otherimplementations are within the scope of the following claims.

Implementations of the various techniques described herein may beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or in combinations of them. Implementations mayimplemented as a computer program product, i.e., a computer programtangibly embodied in an information carrier, e.g., in a machine-readablestorage device (computer-readable medium), for processing by, or tocontrol the operation of, data processing apparatus, e.g., aprogrammable processor, a computer, or multiple computers. Thus, acomputer-readable storage medium can be configured to store instructionsthat when executed cause a processor (e.g., a processor at a hostdevice, a processor at a client device) to perform a process.

A computer program, such as the computer program(s) described above, canbe written in any form of programming language, including compiled orinterpreted languages, and can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program can bedeployed to be processed on one computer or on multiple computers at onesite or distributed across multiple sites and interconnected by acommunication network.

Method steps may be performed by one or more programmable processorsexecuting a computer program to perform functions by operating on inputdata and generating output. Method steps also may be performed by, andan apparatus may be implemented as, special purpose logic circuitry,e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the processing of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Elements of a computer may include atleast one processor for executing instructions and one or more memorydevices for storing instructions and data. Generally, a computer alsomay include, or be operatively coupled to receive data from or transferdata to, or both, one or more mass storage devices for storing data,e.g., magnetic, magneto-optical disks, or optical disks. Informationcarriers suitable for embodying computer program instructions and datainclude all forms of non-volatile memory, including by way of examplesemiconductor memory devices, e.g., EPROM, EEPROM, and flash memorydevices; magnetic disks, e.g., internal hard disks or removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in special purposelogic circuitry.

To provide for interaction with a user, implementations may beimplemented on a computer having a display device, e.g., a cathode raytube (CRT), a light emitting diode (LED), or liquid crystal display(LCD) monitor, for displaying information to the user and a keyboard anda pointing device, e.g., a mouse or a trackball, by which the user canprovide input to the computer. Other kinds of devices can be used toprovide for interaction with a user as well; for example, feedbackprovided to the user can be any form of sensory feedback, e.g., visualfeedback, auditory feedback, or tactile feedback; and input from theuser can be received in any form, including acoustic, speech, or tactileinput.

Implementations may be implemented in a computing system that includes aback-end component, e.g., as a data server, or that includes amiddleware component, e.g., an application server, or that includes afront-end component, e.g., a client computer having a graphical userinterface or a Web browser through which a user can interact with animplementation, or any combination of such back-end, middleware, orfront-end components. Components may be interconnected by any form ormedium of digital data communication, e.g., a communication network.Examples of communication networks include a local area network (LAN)and a wide area network (WAN), e.g., the Internet.

While certain features of the described implementations have beenillustrated as described herein, many modifications, substitutions,changes and equivalents will now occur to those skilled in the art. Itis, therefore, to be understood that the appended claims are intended tocover all such modifications and changes as fall within the scope of theimplementations. It should be understood that they have been presentedby way of example only, not limitation, and various changes in form anddetails may be made. Any portion of the apparatus and/or methodsdescribed herein may be combined in any combination, except mutuallyexclusive combinations. The implementations described herein can includevarious combinations and/or sub-combinations of the functions,components and/or features of the different implementations described.

What is claimed is:
 1. A method, comprising: generating, by a headmounted electronic device operating in an ambient environment, a virtualenvironment; generating, within a display of a handheld electronicdevice operating in the ambient environment, a mono-chromatic screenoverlaid on a user interface within the display of the handheldelectronic device, while maintaining touchscreen functionality of theuser interface; detecting, by a camera of the head mounted electronicdevice, a physical object in a foreground of the mono-chromatic screen;determining, by a processor of the head mounted electronic device, aposition of the detected physical object relative to the user interfacebased on a detected position of the physical object in the foreground ofthe mono-chromatic screen; and displaying, by the head mountedelectronic device, a virtual image of the user interface and a virtualimage of the detected physical object on a display of the head mountedelectronic device.
 2. The method of claim 1, generating a mono-chromaticscreen overlaid on a user interface within the display of the handheldelectronic device, while maintaining touchscreen functionality of theuser interface including: rendering a green screen; and overlaying thegreen screen on a keyboard displayed on the display of the handheldelectronic device.
 3. The method of claim 2, detecting a physical objectin a foreground of the mono-chromatic screen and determining a positionof the physical object relative to the user interface including:detecting a position of a finger against the green screen; and mappingthe detected position of the finger against the green screen to aposition of the finger relative to the keyboard displayed on the displayof the handheld electronic device and overlaid by the green screen basedon a known arrangement of keys of the keyboard displayed on the displayof the handheld electronic device.
 4. The method of claim 3, furthercomprising: receiving a character input in response to a detectedphysical contact between the finger and one of the keys of the keyboarddisplayed on the display of the handheld electronic device and overlaidby the green screen; and displaying the received character input in avirtual display generated by the head mounted electronic device.
 5. Themethod of claim 4, further comprising repeating the detecting a physicalobject, determining a position of the physical object, displaying avirtual image of the user interface and a virtual image the physicalobject, receiving a character input and displaying the receivedcharacter input until a character input mode is terminated.
 6. Themethod of claim 1, detecting a physical object in a foreground of themono-chromatic screen including: activating the camera of the headmounted electronic device; capturing an image of the handheld electronicdevice including the physical object and the mono-chromatic screen usingthe camera of the head mounted electronic device; and separating animage of the physical object from the mono-chromatic screen surroundingthe physical object.
 7. The method of claim 6, displaying a virtualimage of the user interface and a virtual image of the detected physicalobject on a display of the head mounted electronic device including:displaying the separated image of the detected physical object at aposition relative to the virtual image of the user interface tocorrespond to an actual position of the detected physical objectrelative to the user interface underlying the mono-chromatic screen. 8.The method of claim 7, displaying the separated image of the detectedphysical object including displaying the separated image as a passthrough image from the camera on the display of the head mountedelectronic device.
 9. The method of claim 7, displaying the separatedimage of the detected physical object including: rendering the separatedimage of the detected physical object; and displaying the rendered imageof the separated image of the detected physical object on the display ofthe head mounted electronic device.
 10. The method of claim 1, whereinthe detected physical object is positioned between the head mountedelectronic device and the handheld electronic device.
 11. A system,comprising: a head mounted electronic device including a display, acamera and a processor, the head mounted electronic device configured tobe operably coupled with a handheld electronic device, wherein thecamera is configured to capture an image of the handheld electronicdevice, and the processor is configured to: detect a physical objectbetween the head mounted electronic device and a user interface on thehandheld electronic device in the image captured by the camera, the userinterface being overlaid by a mono-chromatic screen, and display avirtual image in a virtual scene generated by the head mountedelectronic device based on the image captured by the camera, the virtualimage displayed in the virtual scene including a virtual image of thedetected physical object, and a virtual image of the user interface ofthe handheld electronic device.
 12. The system of claim 11, wherein thedetected physical object is a finger of a user, and the user interfaceis a keyboard overlaid by the green screen displayed on a display of thehandheld electronic device, and wherein the virtual image displayed inthe virtual scene generated by the head mounted electronic deviceincludes a virtual image of the finger of the user hovering over one ofa plurality of keys of the keyboard on the display of the handheldelectronic device.
 13. The system of claim 12, wherein the processor isconfigured to determine a position of the finger relative to thekeyboard on the display of the handheld electronic device based on thedetected position of the finger relative to the green screen and a knownarrangement of the plurality of keys of the keyboard on the display ofthe handheld electronic device.
 14. The system of claim 12, wherein theprocessor is configured to receive a character input in response tocontact between the finger and a portion of the display corresponding toone of the keys of the keyboard on the display of the handheldelectronic device, and to display a virtual image of the character inputin the virtual image displayed in the virtual scene generated by thehead mounted electronic device.
 15. The system of claim 11, wherein, indisplaying a virtual image in a virtual scene generated by the headmounted electronic device, the processor is configured to: segment animage of the detected physical object from the image captured by thecamera, and display the segmented image of the detected physical objectat a position relative to the virtual image of the user interface tocorrespond to an actual position of the detected physical objectrelative to the user interface underlying the mono-chromatic screen. 16.The system of claim 15, wherein, in displaying the segmented image ofthe detected physical object, the processor is configured to display theseparated image of the detected physical object as a pass through imagefrom the camera in the virtual scene generated by the head mounteddisplay device.
 17. The system of claim 15, wherein, in displaying thesegmented image of the detected physical object, the processor isconfigured to: render an image of the segmented image of the detectedphysical object; and display the rendered image of the segmented imageof the detected physical object in the virtual scene generated by thehead mounted display device.
 18. The system of claim 15, wherein theprocessor is configured to display the virtual image of the userinterface without the mono-chromatic screen overlaid on the userinterface, such that individual elements of the user interface arevisible in the virtual scene displayed by the head mounted displaydevice.
 19. A computer program product embodied on a non-transitorycomputer readable medium, the computer readable medium having storedthereon a sequence of instructions which, when executed by a processorcauses the processor to execute a method, the method comprising:generating a virtual environment in a head mounted electronic deviceoperating in an ambient environment; capturing, by a camera of the headmounted electronic device, an image of a handheld electronic deviceoperably coupled with the head mounted electronic device; detecting, inthe image captured by the camera, a physical object in a foreground of amono-chromatic screen overlaid on a user interface on the handheldelectronic device, the user interface maintaining functionality beneaththe mono-chromatic screen; separating the detected physical object fromthe image captured by the camera and generating an image of theseparated physical object; detecting a position of the detected physicalobject relative to the user interface based on a detected position ofthe physical object in the foreground of the mono-chromatic screen; anddisplaying a virtual image in the virtual environment generated by thehead mounted electronic device, the virtual image including a virtualimage of the user interface and a virtual image of the separatedphysical object separated from the image captured by the camera.
 20. Thecomputer program product of claim 19, displaying a virtual image in thevirtual environment generated by the head mounted electronic devicefurther comprising: displaying the separated image of the detectedphysical object as a pass through image from the camera on the displayof the head mounted electronic device.